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FOREWORD 


Because  the  demand  for  scientific  and  technological  talent  exceeds  the  resources 
available  within  the  Navy  R&D  centers  and  laboratories,  the  Commander  of  the  Space 
and  Naval  Warfare  Systems  Command  (COMSPAWAR)  desires  a  system  managers  can 
use  to  assist  them  in  prioritizing  and  accepting  R&D  projects.  The  Laboratory  Workload 
Prioritization  and  Acceptance  System  (LAWPAS)  was  proposed  and  evaluated  at  the 
Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCEN).  The  results 
of  this  process  are  reported  here. 

Appreciation  is  expressed  to  all  senior  and  middle  managers  and  principal 
investigators  from  NAVPERSRANDCEN  who  were  involved  in  development  of 
LAWPAS.  We  also  express  appreciation  to  Paul  DeYoung  for  assistance  with  the  data 
analysis. 


B.E.  BACON 
Captain,  U.S.  Navy 
Commanding  Officer 


JAMES  S.  McMICHAEL 
Technical  Director 


SUMMARY 


Problem 

Senior  executives  in  the  Space  and  Naval  Warfare  Systems  Command  (SPAWAR) 
expressed  concern  about  the  R&D  project  acceptance  process  in  the  Navy  R&D  centers 
and  laboratories.  A  system  to  assist  R&D  managers  in  evaluating  and  prioritizing 
potential  projects  is  needed.  With  such  a  system,  R&D  managers  would  be  able  to 
evaluate  the  merit  of  potential  projects,  accept  projects  of  highest  priority,  and  defend 
their  choices  to  outside  reviewers.  Such  a  system  would  also  increase  the  congruence  of 
R&D  goals  throughout  the  organization. 

Purposes 

The  purposes  of  this  project  were  to:  (1)  develop  a  method  for  capturing  the 
project  acceptance  policy  of  Navy  R&D  managers;  (2)  develop  a  "predicted  merit 
equation"  based  on  this  policy  using  a  sample  of  active  R&D  projects;  (3)  evaluate  the 
usefulness  of  this  equation  for  aiding  project  acceptance  decisions  and  establishing  a 
work  acceptance  policy. 

Method 

Ratings  of  a  random  sample  of  projects  active  in  FY86  were  collected  from 
attribute  raters  (middle  managers)  and  merit  raters  (senior  managers).  Middle  managers 
used  archival  information  and  descriptions  of  projects  provided  by  principal 
investigators  to  complete  rating  forms  designed  for  the  Laboratory  Workload 
Prioritization  and  Acceptance  System  (LAWPAS).  Senior  managers  used  a  one-page 
project  description  (Form  1498)  to  rank  order  projects  by  overall  merit.  Senior 
managers  were  also  asked  what  criteria  they  believed  were  important  for  evaluating 
potential  R&D  projects  at  the  Navy  Personnel  Research  and  Development  Center 
(NAVPERSR  ANDCEN). 

Results 

Project  information,  attribute  and  merit  raters’  data,  and  multiple  regression 
analyses  are  presented.  The  high  reliability  of  the  ratings  allowed  for  the  formation  of 
regression  equations  with  weighted  dimensions  for  predicting  potential  merit  of  future 
projects. 

Two  equations  were  developed.  The  first  equation  was  based  on  senior  managers' 
ratings  of  overall  merit.  Important  dimensions  for  predicting  merit  by  this  equation 
concerned  whether  the  project  focused  on  issues  central  to  NAVPERSRANDCEN’s 
mission,  whether  it  had  significant  funding  and  was  of  funding  type  6.3  (Advanced 
Development),  and  whether  it  focused  on  Navy  issues.  The  second  equation  was  based 
on  middle  managers’  ratings  of  benefit/cost.  Important  dimensions  in  this  equation  are 
again  concerned  with  whether  the  project’s  focus  was  central  to  NAVPERSRANDCEN's 
mission,  whether  the  project's  end-products  were  likely  to  "transition"  to  other  R&D 
projects  or  the  field,  and  whether  the  project  was  supported  with  6.3  (Advanced 
Development)  funding. 
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INTRODUCTION 


Problem 

Senior  executives  in  the  Space  and  Naval  Warfare  Systems  Command  (SPAWAR) 
expressed  concern  about  the  project  acceptance  process  in  the  Navy  R&D  centers  and 
laboratories.  A  system  to  assist  R&D  managers  in  evaluating  and  prioritizing  potential 
projects  is  needed.  With  such  a  system  R&D  managers  would  be  able  to  predict  the 
merit  of  potential  projects,  accept  projects  of  highest  priority,  and  defend  their  choices 
to  outside  reviewers.  Such  a  system  would  also  increase  the  congruence  of  R&D  goals 
throughout  the  organization. 

Purposes 

The  purposes  of  this  project  were  to:  (1)  develop  a  method  for  capturing  the 
project  acceptance  policy  of  Navy  R&D  managers;  (2)  develop  a  "predicted  merit 
equation"  based  on  this  policy  using  a  sample  of  active  R&D  projects;  (3)  evaluate  the 
usefulness  of  this  equation  for  aiding  project  acceptance  decisions  and  establishing  a 
work  acceptance  policy. 

Background 

Navy  R&D  centers  and  laboratories  provide  the  Navy  community  with  in-house 
R&D  relevant  to  solving  operational  problems.  The  R&D  centers  and  labs  are  a 
significant  asset  in  implementing  our  national  maritime  strategy.  They  have  their 
antecedents  in  the  Navy’s  development  and  testing  organizations  established  during  the 
last  half  of  the  nineteenth  and  early  part  of  the  twentieth  centuries.  There  now  exist  12 
Navy  R&D  centers  and  labs,  each  supporting  a  different  area  of  research  expertise. 

They  "...are  involved  in  all  phases  of  research  and  technology  and  in  all  phases  of  the 
acquisition  and  maintenance  process  from  concept  formulation  to  in-service  Fleet 
support"  (Chief  of  Naval  Material,  1980).  They  are  devoted  to  providing  the  nation  with 
"the  best,  most  up-to-date,  capable,  and  effective  Fleet  and  Marine  Corps  forces  which 
modern  technology  can  provide  for  the  resources  available"  (Department  of  the  Navv, 
1985). 


In  the  past,  the  Navy  R&D  centers  and  labs  have  operated  under  normal 
constraints  concerning  the  hiring  of  personnel,  the  contracting  out  of  projects,  and  the 
distribution  of  funds  from  one  fiscal  year  to  the  next.  The  hope  has  been  that  the 
constraints  would  be  short-lived,  allowing  the  center  and  labs  to  expand  to  meet  the 
demands  of  the  available  workload.  Unfortunately  due  to  increasingly  constrained 
federal  fiscal  resources,  there  is  no  reason  to  believe  Navy  R&D  resources  will  expand. 

It  is  expected  that  the  present  constraints  will  increase,  thus  limiting  the  amount  of  work 
the  centers  and  labs  can  assume. 

These  constraints  have  forced  managers  to  reevaluate  methods  for  achieving  their 
organizational  goals.  One  method  for  doing  this  is  to  accept  only  projects  that  are  of 
highest  priority  and  payoff  to  the  organization.  Thus  the  issue  of  specifying  a  project 
acceptance  policy  and  process  becomes  critical. 

Hundreds  of  project  evaluation  and  selection  models  have  been  proposed  and 
tested.  Very  few  have  been  systematically  implemented.  Costello  (1983)  provides  a 
review  of  many  of  these  models,  their  methods  and  their  shortcomings.  Williams  (1969) 
has  categorized  the  models  into  three  types  based  on  their  approach:  (1)  decision  theon. 
(2)  economic  analysis,  and  (3)  operational  research.  He  concludes  that  the  decision 
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theory  approach  is  optimal,  for  it  allows  the  use  of  numerous  evaluation  factors 
simultaneously. 

The  decision  theory  approach  asks  decision  makers  to  identify  critical 
dimensions,  factors,  or  criteria  for  evaluating  potential  R&D  projects.  Many  research 
efforts  have  been  concerned  with  these  dimensions  (Costello,  1983).  The  Defense 
Science  Board  (Department  of  Defense,  1981)  suggests  three  such  criteria: 

1.  Impact  of  opportunity.  Mission  value,  technology  impact  on  mission, 
pervasiveness  of  impact. 

2.  Technical  risks.  Maturity  of  technology,  innovation  potential. 

3.  System/operational  concept  risks.  Mission  risk,  political  environment,  level 
of  operational  support. 

Doherty  and  Seymour  (1983)  suggest  seven  dimensions: 

1.  Importance /necessity.  Value  of  the  project  to  the  Navy. 

2.  Payoff.  Value  of  the  product  to  the  Navy  weighted  against  the  resources 
required  for  R&D  and  implementation. 

3.  Timeliness.  Urgency  to  complete  the  project,  political  requirements,  etc. 

4.  Probability  of  success.  Probability  that  the  effort  would  be  successful  (not 
including  the  probability  that  results  could  be  implemented). 

5.  Validity  as  R&D.  Scientific  research  and  development. 

6.  Link  to  other  work.  Planned  or  approved  work  dependent  upon  completion  of 
this  project. 

7.  Implementation.  The  difficulty  of  implementing  project  end-products. 

In  the  decision  theory  approach,  dimensions  are  identified  and  weights  attached 
to  the  dimensions  to  allow  the  scoring  of  R&D  projects  (Williams,  1969).  These  scores 
can  then  be  ranked,  and  the  highest  scoring  projects  accepted  based  on  available 
resources.  Thus  by  identifying  and  appropriately  weighting  critical  dimensions  for 
evaluating  R&D  efforts,  managers  may  be  able  to  predict  the  merit  of  potential  work 
and  thereby  decide  whether  or  not  to  pursue  the  effort.  This  process  aids  decisiuii 
making  under  conditions  of  uncertainty  (Costello,  1983). 

Besides  the  optimizing  of  resources,  other  benefits  are  likely  to  result  from 
systematic  R&D  project  evaluation  and  acceptance  procedures.  An  explicit  statement  by 
upper-level  managers  of  what  makes  projects  valuable  clarifies  for  others  the  R&D  goals 
of  the  organization.  The  values,  judgments,  and  standards  of  senior  management  can 
thus  be  applied  to  project  evaluation  and  acceptance  decisions  without  senior  managers' 
involvement  in  every  decision  (European  Industrial  Research  Management  Association. 
1978).  Having  a  systematic  process  of  project  selection  also  simplifies  explaining  R&D 
project  portfolios  to  outside  reviewers.  The  process  can  encourage  communication 
between  different  groups  within  the  organization,  and  assist  the  organization  in  reaching 
consensus  concerning  its  goals  (Sounder,  1975). 


Ideally,  the  development  of  a  Laboratory  Workload  Prioritization  and  Acceptance 
System  (LAWPAS)  would  result  in  the  following: 

1.  Senior  and  middle  managers’  ideas  regarding  the  relevant  dimensions  and 
weights  for  determining  a  project’s  merit  would  be  captured  in  the  policy  equation. 

2.  Researchers  could  implement  managers’  policies  in  an  economical  and 
decentralized  manner. 

3.  Once  made  explicit,  the  policy  equation  could  be  systematically  evaluated  and 
revised  as  necessary. 

4.  Projects  could  be  evaluated  using  the  policy  equation  at  midyear,  year-end.  or 
milestone  points. 

5.  If  desired,  the  policy  equations  and  minimum  priority  values  or  "cut  scores" 
used  to  accept  work  could  vary  across  different  departments  as  well  as  across  time  as 
workload  demands  and  resource  availability  fluctuate. 

6.  The  whole  process  of  work  acceptance  would  be  more  systematic. 

7.  Increased  discussion  among  organzational  members  concerning  R&D  goals  and 
means  to  achieving  these  goals  would  result. 

METHOD 

The  methodology  for  developing  a  Navy  R&D  center  workload  prioritization  and 
acceptance  system  was  tested  at  the  Navy  Personnel  Research  and  Development  Center 
(NAVPERSRANDCEN)  during  FY87.  Center  managers  identified  critical  dimensions 
for  the  NAVPERSRANDCEN  policy  equation.  The  following  eight  steps  summarize  the 
procedure  used  in  LAWPAS: 

1.  Identification  of  dimensions  for  evaluating  R&D  proposals. 

The  initial  step  was  to  establish  a  pool  of  dimensions  for  evaluating  potential 
work  at  NAVPERSRANDCEN.  Examples  of  dimensions  critical  in  the  R&D  selection 
process  were  derived  from  past  efforts  devoted  to  characterizing  R&D  projects  and 
products  (Costello,  1983).  In  addition,  senior  managers  were  asked  to  provide 
dimensions  they  believed  to  be  important  in  the  R&D  selection  and  evaluation  process. 
The  final  pool  of  dimensions  included  those  identified  above  plus  the  following  unique 
items  provided  by  NAVPERSRANDCEN  senior  managers: 

a.  Snonsorship.  The  organizational  level  or  rank  of  the  project  sponsor  and,  or 
endorser. 

b.  Funding  level.  The  amount  and  number  of  years  of  funding  for  the  project. 

c.  Type  of  Funding.  Research  and  development  program  element  funds  (6.1  - 
6.5)  versus  reimbursable  funds  (program  element  code  number  99000). 

A  project  rating  form  having  29  items  based  upon  the  attributes  of  these 
dimensions  was  developed  (Appendix  A). 


2.  Selection  of  a  sample  of  research  projects. 

A  random  sample  of  projects  was  used  for  data  collection.  The  number  of 
projects  selected  from  each  department  was  proportional  to  the  number  of  projects 
funded  in  the  department  for  FY86.  Sixty-three  projects  were  selected  out  of  178  R&D 
projects  funded  in  FY86.  Contracted  projects  and  projects  in  support  of  R&D  efforts 
were  not  chosen. 

A  principal  investigator  (PI)  questionnaire  (Appendix  B)  was  sent  out  to  all  Pis 
for  the  selected  projects  requesting  project  information.  This  information  was  both 
factual  (i.e.,  funding  level,  sponsors/endorser,  start/end  dates,  etc.)  and  perceptual 
(impact  of  end-products,  personnel  affected  by  end-products,  etc.). 

i.  Collection  of  project  attributes  ratings  from  middle  managers. 

Eighteen  middle  managers  (GM/GS-13s  and  GM-14s)  nominated  by  senior-level 
managers  served  as  attribute  raters.  Senior  managers  were  asked  to  select  as  raters 
managers  with  a  broad  range  of  experience  and  exposure  to  NAVPERSR ANDCEN  work. 
The  number  of  attribute  raters  for  each  department  was  proportional  to  the  number  of 
people  in  each  department. 

Each  attribute  rater  rated  a  group  of  14  projects.  The  assignment  of  projects  to 
groups,  and  groups  to  raters  was  random,  except  that  in  no  case  was  an  attribute  rater 
asked  to  rate  his/her  own  project.  The  grouping  and  assignment  were  done  in  a  way 
that  guaranteed  that  each  project  was  rated  four  times,  once  each  by  four  different 
raters.  This  procedure  was  chosen  to  minimize  the  workload  required  of  the  attribute 
raters,  yet  allow  reliable  estimates  to  be  generated  for  each  project  by  all  raters. 

A  meeting  was  held  with  all  of  the  attribute  raters  to  describe  the  project  and 
discuss  how  to  complete  the  questionnaires.  Attribute  raters  received  the  completed 
principal  investigator  questionnaires  and  Form  1498s  (Appendix  C)  and  other 
documentation  [Technical  Development  Plans  (TDPs)  and  Plan  of  Action  and  Milestones 
(POA&M)]  for  all  the  projects  they  were  asked  to  rate.  Attribute  raters  completed  one 
attribute  questionnaire  for  each  of  the  14  projects. 

4.  Collection  of  overall  merit  ratings  from  senior  managers. 

Senior  managers  (Commanding  Officer.  Technical  Director,  Department 
Directors,  and  other  GM-15s)  rated  the  selected  projects  on  their  overall  merit,  taking 
into  consideration  the  costs  in  pursuing  the  project  and  the  benefits  that  would  ensue. 
This  procedure  was  designed  to  establish  the  relative  merit  of  the  projects  as  seen  by 
these  senior  managers. 

A  comparison  procedure  was  used  to  obtain  the  overall  merit  ratings.  Three 
projects  were  eliminated  prior  to  this  rating  process  because  upon  closer  examination 
they  were  found  to  be  contractual  efforts  that  did  not  involve  research  and  development 
One-page  descriptions  (Form  1498)  for  the  60  projects  were  placed  in  a  stack  in  front  of 
the  merit  rater.  The  rater  was  asked  to  sort  the  projects  in  one  of  two  ways:  (1)  The 
rater  could  use  the  first  project  as  the  anchor  for  the  middle  category,  and  place  all 
projects  into  categories  of  lower,  equivalent,  or  higher  merit  compared  to  this  first 
project;  or  (2)  the  rater  could  separate  the  projects  into  three  merit  categories  of  low. 
medium,  and  high.  The  merit  rater  used  whichever  procedure  was  personally  most 
meaningful. 
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Upon  separating  the  60  projects  into  three  categories,  each  rater  then  separated 
the  projects  within  each  category  again,  using  the  same  procedure.  This  process  was 
continued  until  the  merit  rater  felt  the  projects  were  adequately  sorted  and  could  no 
longer  separate  the  projects  into  additional  categories.  Categories  of  projects  were 
assigned  ranks,  with  all  the  projects  that  were  in  the  same  category  receiving  the  same 
number.  Projects  with  the  lowest  merit  received  the  rank  of  one. 

After  completing  the  sorting  procedure,  merit  raters  were  asked  what  dimensions 
they  felt  were  important  in  evaluating  the  merit  of  potential  R&D  projects  at 
NAVPERSRANDCEN. 

5.  Estimation  of  attribute  raters’  ratings  for  all  items  for  all  projects. 

The  Ward-Jennings  technique  (1979)  was  used  to  estimate  the  attribute  raters’ 
ratings  for  those  projects  they  did  not  rate.  This  procedure  estimated  the  rating  of  all 
projects  as  if  the  raters  had  rated  all  projects  on  all  29  items.  It  provided  a  complete  set 
of  ratings  for  all  projects  by  all  raters,  without  having  to  require  that  each  rater 
complete  63  questionnaires. 

6.  Creation  of  a  policy  equation. 

The  policy  of  senior  managers  in  assigning  overall  merit  to  the  projects  was 
captured  through  the  use  of  a  least  squares  fit  to  a  general  linear  model.  Ratings  on  the 
dimensions  made  by  the  attribute  raters  along  with  objective  information  about  the 
project  were  statistically  regressed  on  the  project’s  average  overall  merit  rating  to 
determine  what  dimensions  were  important  in  estimating  merit,  and  the  relative  weights 
associated  with  each  dimension.  The  solution  to  this  equation  made  explicit  which 
dimensions  senior  managers  believe  make  a  project  successful  at  NAVPERSRANDCEN. 
These  same  procedures  were  used  to  capture  the  policy  reflected  in  attribute  raters’ 
estimates  of  the  overall  benefit/cost  ratio  of  the  projects. 


RESULTS 


The  distribution  of  projects  and  raters  by  department  is  presented  below. 

Table  1 

DISTRIBUTION  OF  LAWPAS  PROJECTS  AND  RATERS 

Department  Merit  Attribute  Projects 

Raters  Raters  Selected 

00/01  2 

31  1 

41  1  6  18 

51  2  2  6 

52  2  4  14 

61  1  2  7 

62  1  3  15 

63  1  13 

Total  11  18  63 

The  results  will  be  presented  in  several  sections:  (1)  project  data,  (2)  attribute 
rater  data,  (3)  merit  rater  data,  and  (4)  building  the  policy  equation. 

Project  Data 

The  principal  investigator  (PI)  questionnaires  (Appendix  B)  and  Form  1498 
(Appendix  C)  provided  detailed  information  about  the  63  projects  selected.  Projects 
with  several  different  types  of  funding  appeared  in  the  random  sample.  Table  2 
summarizes  the  projects  by  funding  program  element. 


Table  2 


FUNDING  CATEGORIES  FOR  SELECTED  PROJECTS 

Funding  Category  Number  of  Projects 


61152N 

In-house  Independent  Research 

62744N 

Marine  Corps  Air  Ground  Technology 

*> 

62757N 

Human  Factors  and  Simulation  Technology 

3 

62763N 

Personnel  and  Training  Technology 

8 

63502N 

Advanced  Surface  Mine  Countermeasures 

1 

6370  IN 

Human  Factors  Engineering 

63707N 

Manpower  and  Personnel  Systems 

3 

63720N 

Education  and  Training 

3 

63732N 

Advanced  Manpower/Training 

4 

63733N 

Simulation  and  Training  Devices 

1 

63739N 

Navy  Logistics  Productivity 

1 

64722A 

Education  and  Training  System  Development  -  Army 

1 

99000N 

Reimbursables 

32 

Note:  Only  the  primary  funding  category  is  reported. 


63 


As  can  be  seen,  the  projects  were  evenly  divided  between  R&D  and 
reimbursable-funded  projects.  The  mean  FY86  funding  level  for  the  selected  projects 
was  $210,546;  the  median  was  $120,500,  and  standard  deviation  (SD)  was  $233,308. 

A  summary  of  the  sponsors  and  endorsers  of  the  projects  is  presented  in  Table  3 
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Table  3 


PROJECT  SPONSORS/ENDORSERS  AS  REPORTED 
BY  THE  PRINCIPAL  INVESTIGATORS 

Sponsoring  Command  Number  of  Projects * 

Secretary  of  the  Navy/ 

Assistant  Secretary  of  the  Navy  6 

Department  of  Defense  7 

Office  of  the  Chief  of  Nava!  Operations  21 

Office  of  Naval  Research/Office  of  Naval 

Technology/Chief  of  Naval  Research  18 

Space  and  Naval  Warfare  Systems  Command/ 

Naval  Sea  Systems  Command/Naval  Air  Systems 
Command/Naval  Supply  Systems  Command/Navy 


Data  Automation  Command  18 

Navy  Military  Personnel  Command/ 

Navy  Civilian  Personnel  Command  8 

Chief  of  Naval  Education  and  Training/ 

Chief  of  Naval  Technical  Training  7 

Other  22 


*  Does  not  sum  to  63  because  multiple  responses  were  permitted. 


Project  end-products  usually  were  in  the  form  of  briefings,  technical 
reports/notes,  software,  and  conference  papers.  The  mean  number  of  months  to  deliver 
end-products  was  18.33,  with  a  median  of  12.00,  and  standard  deviation  of  18.91.  The 
settings  in  which  end-products  are/will  be  used  are  summarized  in  Table  4. 

Attribute  Rater  Data 

Each  of  the  18  attribute  raters  completed  the  attribute  questionnaire  (Appendix 
A)  for  a  random  set  of  14  projects.  The  Ward-Jennings  (1979)  technique  was  used  to 
estimate  attribute  raters’  ratings  for  all  projects  not  rated  on  the  29  items.  The  multiple 
regression  coefficients  from  the  analyses  used  to  estimate  the  unobserved  data  elements 
indicate  the  degree  of  agreement  between  raters  and  the  accuracy  of  the  estimation  of 
ratings  not  made.  The  multiple  R  ranged  from  .71  to  .99;  the  mean  R  was  .81.  These 
values  indicate  a  high  degree  of  agreement  among  raters  on  the  project  attributes.  The 
analysis  procedure  made  it  possible  to  correct  for  rater  bias  in  the  ratings  given.  With  a 
corrected  data  set,  the  corrected  average  across  raters  for  each  of  the  29  items  for  all  the 
projects  was  computed. 
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Table  4 


THE  VARIOUS  SETTINGS  WHERE  PROJECT  END-PRODUCTS 

WILL  BE  USED 


Setting 


Number  of  Projects * 


Navy  Shore  Activities 
Navy  Operational  Fleet 
Navy  R&D  Centers/Laboratories 
Other  Military  Activities 
Government  or  DoD  Headquarters 


Universities 


Private  Sector  Business 


Navy  Industrial  Activities 
Other  Settings 


*  Does  not  sum  to  63  because  multiple  responses  were  permitted. 


Next,  a  principal  components  factor  analysis  of  the  29  average  corrected 
attributes  was  performed.  A  varimax  rotation  resulted  in  six  dimensions,  which  are 
described  in  Table  5.  Alpha  coefficients  were  computed  to  estimate  the  reliability  of  the 
composite  scores  created  by  summing  the  items  that  clustered  together  in  the  factor 
analysis.  The  alpha  coefficients  ranged  between  .72  and  .91.  The  means,  standard 
deviations,  and  alpha  coefficients  are  showm  in  Appendix  D,  Table  D-l. 

Merit  Rater  Data 

The  merit  raters  ranked  the  overall  merit  of  the  projects.  Ties  were  permitted. 
Sixty  of  the  projects  received  an  overall  merit  rank  score  based  on  the  merit  rater 
sorting  procedure.  These  scores  were  then  standardized  within  raters  to  control  for  the 
number  of  categories  into  which  the  projects  had  been  sorted,  since  this  number  varied 
somewhat  by  rater.  The  average  merit  rating  for  each  project  was  then  calculated  across 
all  1 1  merit  raters.  Again,  the  degree  of  agreement  between  raters  was  analyzed.  The 
alpha  coefficient  for  interrater  agreement  was  .86,  showing  a  high  degree  of  agreement 
between  raters. 


Table  5 


QUESTIONNAIRE  DIMENSIONS  POTENTIALLY  RELATED  TO  PROJECT  MERIT 

1.  MISCNTRD  Centrality  to  NPRDC's  Mission 

Questionnaire  items 1 

Ala.  To  what  extent  are  the  iuues  addressed  by  this  project  important  to  NPRDC? 

A2  To  what  extent  do  the  issues  addressed  by  this  project  correspond  with  NPRDC’s  formal  mission? 

Bla  To  what  extent  does  this  project  build  upon  NPRDC  research  and  development? 

B2a  To  what  extent  does  this  project  integrate  with  current  NPRDC  projects? 

Cl  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  to  the  R&D  community? 

C3  To  what  extent  will  the  end-product(s)  from  this  project  reflect  positively  on  NPRDC? 

C5.  To  what  extent  will  this  work  facilitate  future  opportunities  for  funding  and/or  applications? 

CCd  To  what  extent  is/will  this  project,  compared  to  other  NPRDC  projects,  be  import  ant  and  valuable  to  the  Navy  by  expat; -ling 
the  technical  base  (developing  new  technology,  procedures,  etc)? 

2.  DODRD  Connectivity  with  DOD  R&D 

Questionnaire  items 

Blc  To  what  extent  does  this  project  build  upon  other  DoD  research  and  development? 

B2c  To  what  extent  does  this  project  integrate  with  current  DoD  R&D  projects? 

3.  TRANSITN  Likelihood  of  Transitioning 

Questionnaire  items 

B3c  To  what  extent  does  the  approach  for  this  project  seem  to  be  feasible? 

C4  What  is  the  probability  that  end-products  from  this  project  could  be  implemented'' 

4.  RISK  Project  Risk 

Questionnaire  items 

A3  To  what  extent  is  NPRDC  uniquely  qualified  to  do  this  work?  (reverse  scored) 

Bib.  To  what  extent  does  this  project  build  upon  other  Navy  research  and  development? 

B2b  To  what  extent  does  this  project  integrate  with  current  R&D  projects? 

B4a.  To  what  extent  are  there  administrative  risks  associated  with  pursuing  this  project7 
B4b  To  what  extent  are  there  technical  risks  associated  with  pursuing  this  project? 

5.  NAVYOP  Usefulness  to  Operational  Navy 

Questionnaire  items 

Aid  To  what  extent  are  the  issues  addressed  by  this  project  important  to  the  operational  Navy7 
A2  To  what  extent  do  this  issues  addressed  by  this  project  correspond  with  NPRDC’s  formal  mission7 
C2  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  for  Navy  applications7 

C6a  To  what  extent  is/will  this  project,  compared  to  other  NPRDC  projects,  be  important  and  valuable  to  the  Navy  by  improving 
Navy  effectiveness  (e  g.,  meeting  operational  requirements,  achieving  stated  goals,  etc.)? 

C6b  To  what  extent  is/will  this  project,  compared  to  other  NPRDC  projects,  be  important  and  valuable  to  the  Navy  by  impr-  -ving 
Navy  efficiency  (e.g.,  improving  use  of  resources,  reducing  costs,  etc.)? 

6.  SOUND  Scientific  validity 

Questionnaire  items 

B3a  To  what  extend  does  the  approach  for  this  project  seem  to  be  methodologically  sound7 
B3b.  To  what  extend  does  the  approach  for  this  project  seem  to  be  theoretically  sound7 

*The  three  major  sections  of  the  questionnaire  are  designated  by  capital  letters  (Appendix  A)  Section  ’’A"  refers  to  A* '  r.:  i-  •••  f  t  b 
Problem  Addressed,  "B"  to  Attributes  of  the  Project,  and  "C"  to  Attributes  of  End-products 
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Upon  completing  the  sorting  technique,  raters  were  asked  what  dimensions  the\ 
were  using  during  the  rating  process.  These  comments  were  r  ntent-analyzed  and 
collapsed  into  five  dimensions;  these  are  presented  in  Table  6  Three  of  the  merit  raters 
were  asked  to  rate  the  projects  again  to  get  a  measure  of  the  intra-rater  reliability.  The 
average  correlation  between  the  original  and  second  rating  was  .77,  ranging  from  .71  to 
.85. 


Table  6 

MERIT  RATERS’  SELF-REPORT  OF  CRITICAL  DIMENSIONS 
FOR  EVALUATING  POTENTIAL  PROJECTS 

Is  it 

Is  there  a  well-articulated  research  goal? 

Will  this  contribute  to  the  technology  base? 

Is  this  project  innovative,  testing  new  ideas? 

Does  this  build  on  prior  work  in  the  area? 

Priority  for  the  operational  forces? 

Does  this  address  an  identified  Navy  priority  (i.e.,  ASW)? 

What  will  the  impact  be? 

Will  this  effort  result  in  products  with  generalized  application  to  the  Navy? 
Does  this  project  have  milestones  with  deliverable  and  usable  products? 

Are  the  end-products  transitionable  to  other  R&D  or  the  fleet? 

Will  the  end-products  impact  a  broad  range  and  number  of  people? 

Is  there  an  identifiable  user? 

Top-level  support? 

Does  this  effort  have  adequate  funding? 

Will  this  project  have  continued  funding  over  time? 

Is  this  project  Congressionallv-mandated? 

Do  we  have  the  resources? 

Is  there  a  better  agency  to  do  the  work? 

What  is  the  reputation  of  the  principal  investigator? 

Is  this  "body  shopping”  that  could  easily  be  done  by  a  contractor? 


Building  the  Policy  Equation 

All  the  variables  from  both  groups  of  raters  and  the  principal  investigator 
information  were  correlated  to  identify  significant  relationships  between  variables 
(Appendix  E).  A  description  of  all  the  variables  with  significant  correlations  is 
presented  in  Table  7.  The  means,  standard  deviations,  and  alpha  coefficients  are  found 
in  Appendix  D. 


Table  7 

VARIABLES  INCLUDED  IN  THE  REGRESSION  ANALYSES 


Merit  Rater  and  Attribute  Rater  Variables 


Variable 


Label 


MERIT 
B  C 

M1SCNTRD 

NAVYOP 

RISK 

DODRD 

TRANSITN 

SOUND 


Average  merit  rating  (CO,  TD,  and  all  GM-15’s) 
Overall  benefit/cost  from  the  attribute  raters 
Central  to  mission  of  the  Center 
Usefulness  to  operational  Navy 
Degree  of  risk 

Connectivity  with  DoD  R&D 
Likelihood  of  transitioning 
Scientific  validity 


■ii  chiral  Information  Variables 


Variable 


Label 


ACTFUND 

LFUND 

TYPE1 

TYPE2 

TYPE3 

TYPE4 

TYPER 


Actual  funds  received  in  FY86 

Log  transform  of  the  FY86  actual  funding 

6.1  Funding  (binary) 

6.2  Funding  (binary) 

6.3  Funding  (binary) 

6.4  Funding  (binary) 

Reimbursable  funds  (binary) 


Principal  Investigator  Questionnaire  \  ’ariables 
Variable  Label 


INTEGRTN 

PERCENT 

YRS 

INHOUSE 

DIFFCLTY 

CONTRCT% 

INF/KNOW 

POLICY 

PROCEDR 

SPONSORS 

SETTINGS 


Degree  of  integration  with  other  projects 

Percent  of  overall  effort 

Years  until  planned  completion 

Percent  in-house  resources  available  at  start-up 

Difficulty  in  getting  resources 

Percent  contracted 

Will  the  end-products  change  information/knowledge?  (binary) 
Will  the  end-products  change  policy?  (binary) 

Will  the  end-products  change  procedures/techniques?  (binary) 
Number  of  sponsors 

Number  of  settings  in  which  products  will  be  used 
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Two  stepwise  regression  analyses  were  conducted.  All  the  variables  in  Table  7 
were  entered  into  the  regression  analysis  to  predict  two  criteria.  The  first  analysis 
regressed  the  variables  on  the  average  merit  rating  calculated  from  the  merit  raters.  The 
results  are  presented  in  Table  D-2.  forming  the  following  equation: 

PMER1T  =  (.690  *  MISCNTRD)  +  (.321  *  LFUN’D)  +  (-.241  *  DODRD) 

+  (-.212  *  POLICY)  +  (.172  *  TYPE3) 

The  second  analysis  regressed  the  variables  on  the  overall  benefit/cost  rating 
produced  from  the  attribute  raters.  Table  D-3  presents  the  results  of  this  analysis  and 
the  following  equation: 

PB/C  =  (.773  *  MISCNTRD)  +  (.378  *  TRANSITN)  +  (.145  *  TYPE3)  +  (-.154  *  RISK) 

+  (-.143  *  INHOUSE) 

The  weights  reported  here  in  the  equations  are  the  standardized  regression 
weights  (Beta).  The  correlation  between  actual  merit  and  actual  benefit/cost  ratings  and 
predicted  merit  and  predicted  benefit/cost  ratings  is  summarized  in  Table  8. 


Table  8 

CORRELATIONS  BETWEEN  MERIT  CRITERIA 
AND  PREDICTED  CRITERIA 


MERIT 

B/C 

B/CMRT 

PMERIT 

PB/C 

B/C 

.550 

1.000 

B/CMRT 

.880 

.880 

1. 000 

PMERIT 

.830 

.702 

.870 

1.000 

PB/C 

.638 

.921 

.886 

.794 

1.000 

PB/CPMRT 

.775 

.857 

.927 

.947 

.947 

Labels 

MERIT  Average  merit  rating 

B/C  Overall  benefit/cost  rating  from  attribute  raters 

B/CMRT  Average  of  MERIT  and  B/C 

PMERIT  Predicted  MERIT 

PB/C  Predicted  B/C 

PB/CPMRT  Average  of  predicted  B/C  and  predicted  MERIT 


CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions  and  recommendations  are  concerned  with  four  areas  based  on  the 
results  described  above:  (1)  interpretation  of  policy  equations,  (2)  benefits  of  LANVPAS. 
(3)  evaluation  of  LAWPAS,  and  (4)  future  applications. 

Interpretation  of  Policy  Equations 

Several  conclusions  can  be  drawn  from  a  review  of  the  dimensions  found  in  the 
policy  equations.  The  equation  based  on  the  merit  raters  emphasizes  several  dimensions. 
Projects  focusing  on  issues  central  to  NA VPERSRANDCEN'S  mission  (MISCNTRD) 
have  higher  worth  and  merit.  Also  6.3  Advanced  Development-funded  projects 
(TYPE3)  with  significant  funding  (LFUND)  were  rated  higher  in  terms  of  merit. 

Projects  addressing  Navy  R&D  concerns  were  rated  higher  than  those  built  upon  or 
integrated  with  DoD  projects  (negative  weight  on  variable  DODRD).  Lastly,  projects 
thought  to  influence  policy  (negative  weight  on  variable  POLICY)  were  not  valued  as 
much  as  those  addressing  information/knowledge  (INF/KNOW)  and 
procedures /techniques  (PROCEDR). 

The  equation  based  on  the  attribute  raters  is  similar  in  several  respects  but 
somewhat  different.  Middle  managers/researchers  agreed  with  senior  managers  that 
projects  addressing  the  center's  mission  were  more  highly  valued  (MISCNTRD).  The\ 
also  valued  projects  that  were  likely  to  transition  (TRANSITN)  to  other  R&D  projects  or 
into  the  field.  Attribute  raters  also  rated  highly  those  projects  with  6.3  (Advanced 
Development)  funding  (TYPE3)  and  projects  with  lower  risk  (negative  weight  on 
variable  RISK).  Also  rated  highly  were  those  projects  perceived  to  be  of  sufficient 
value  to  warrant  redirection  of  in-house  resources  to  make  the  project  possible  (negati\e 
weight  on  the  variable  INHOUSE). 

A  few  rather  unexpected  relationships  appeared  in  these  equations.  Several 
managers  expressed  surprise  at  the  fact  that  DoD  R&D  was  not  highly  valued,  as 
demonstrated  by  the  negative  weight  attached  to  that  dimension.  At  face  value  this 
appears  to  be  contradictory  to  the  mission  of  a  Navy  R&D  center  that  is  part  of  DoD  . 
But  it  was  found  that  those  projects  focusing  on  Navy  issues  (NAVYOP)  were  of  very 
high  merit.  Thus  it  seems  that  projects  focusing  on  issues  useful  to  the  operational  N'a\\ 
are  seen  as  having  greater  merit  than  those  focusing  on  issues  important  to  other 
services.  The  negative  relationship  with  the  policy  variable  also  suggests  an  inverse 
effect,  that  those  projects  impacting  information/knowledge  and  procedure  techniques 
are  seen  as  more  valuable. 

The  negative  relationship  between  the  risk  variable  and  overall  merit  should  not 
be  taken  at  face  value,  for  comments  from  attribute  raters  emphasized  that  they  valued 
risk  when  conducting  R&D.  It  is  realistic  to  say  that  projects  with  high  risk  have 
unknown  payoffs,  and  usually  have  low  expected  benefit/cost  ratings.  Since  this 
equation  was  calculated  in  terms  of  overall  benefit/cost,  it  is  not  surprising  that  the  risk 
variable  was  rated  in  this  fashion. 

These  examples  demonstrate  an  important  point  concerning  LAWPAS.  The 
policy  captured  by  these  equations  is  the  policy  being  used  by  managers  to  evaluate 
existing  or  past  work.  This  captured  policy  may  not  represent  the  preferred  policy. 

W'hat  has  been  reported  in  the  results  are  those  dimensions  that  managers  currently  value 
in  judging  and  accepting  work.  It  is  possible  that  the  organization  would  like  to  shape 
those  values  differently,  and  consciously  emphasize  dimensions  that  may  not  currentb  be 
highly  valued.  Thus  the  policy  equation  may  in  the  future  be  altered  to  reflect  those 
dimensions  the  organization  would  like  to  emphasize  in  its  work  acceptance  polic\.  Tim 


equation  could  be  a  combination  of  the  dimensions  both  middle  and  senior  managers 
value.  This  would  allow  for  the  clear  and  concise  specification  of  the  organization’s 
R&D  goals. 

Benefits  of  LAWPAS 

LAWPAS  is  useful  in  many  respects.  The  process  of  rating  projects  and 
capturing  policy  provides  insight  to  the  organization.  It  clarifies  for  managers  what 
dimensions  they  perceive  to  be  valuable  in  accomplishing  their  organization’s  mission. 
This  process  also  clarifies  the  extent  to  which  the  policy  viewpoints  of  senior  and  middle 
managers  are  congruent. 

The  policy  equation  may  also  be  used  by  managers  in  a  proactive  way  to  design 
projects  emphasizing  those  dimensions  valued  by  the  organization.  Currently,  projects 
lasting  longer  periods  of  time  with  larger  amounts  of  funding  and  emphasizing  Navy 
R&D  concerns  are  more  highly  valued  than  quick  turnaround  projects  with  low  levels  of 
funding  and  focusing  on  services  or  groups  other  than  Navy.  Revising  a  project  to  fit 
this  profile  should  increase  its  merit  score  and  also  better  reflect  the  mission  of  the 
organization. 

Managers  expressed  concern  that  implementing  the  policy  equation  would 
infringe  on  their  decision-making  authority  in  terms  of  work  acceptance.  It  needs  to  be 
clearly  understood  that  the  policy  equation  is  based  upon  data  generated  by  the 
managers.  The  essence  of  their  policy  viewpoints  is  captured  with  the  equation.  The 
predicted  merit  scores  are  just  another  source  of  information  when  deciding  whether  to 
accept  or  reject  a  project.  This  information  may  be  particularly  useful  for  projects  that 
are  difficult  to  decide  upon  in  terms  of  acceptance.  Managers  will  need  to  continue  to 
give  consideration  to  a  variety  of  information  in  the  decision-making  process,  such  as 
the  priority  of  the  requirement  or  whether  the  request  comes  from  a  long-term  sponsor. 
The  results  of  the  policy  equation  will  not  replace  the  manager  in  the  work  acceptance 
decision  but  will  serve  as  a  tool  to  assist  the  manager  in  making  the  decisions  required. 

In  the  past,  simple  work  acceptance  algorithms  have  proven  much  more  effective 
than  complex,  sophisticated  algorithms  (Liberatore  &  Titus,  1983).  Thus,  in  developing 
a  decision  aid,  a  simple  algorithm  that  satisfactorily  deals  with  the  work  acceptance 
problem  should  be  used  to  estimate  the  merit  of  proposed  R&D  projects.  The  LAWPAS 
equation  seems  to  meet  that  criterion.  By  using  the  LAWPAS  equation  to  predict  the 
merit  of  potential  projects,  the  manager  has  an  aid  to  assist  in  the  project  selection  and 
evaluation  process.  If  the  decision  is  not  to  accept  the  project  at  this  time  because  its 
predicted  merit  is  low,  the  manager’s  choices  are  to  (a)  defer  the  decision  and  postpone 
the  effort,  (b)  renegotiate  the  terms  of  the  project  so  that  it  is  of  higher  merit,  or  (c) 
reject  it  outright. 

Evaluation  of  LAWPAS 

There  are  three  areas  where  LAWPAS  needs  to  be  evaluated  as  it  is  implemented 
within  an  organization.  The  high  reliability  achieved  from  18  attribute  raters  and  11 
merit  raters  may  not  be  achieved  when  using  a  somewhat  smaller  number  of  raters.1 


In  the  process  of  collecting  the  ratings  from  both  attribute  and  merit  raters,  many  people  expressed  skepticism 
that  agreement  of  ratings  would  be  found.  Several  attibute  raters  felt  that  they  could  not  rate  the  projects  on  the 
dimensions,  and  that  the  data  resulting  from  this  would  be  random  and  incomprehensible  Interestingly  enough 
much  more  agreement  was  found  between  the  raters  than  even  the  authors  expected  Attribute  raters  agreed  on 
ratings  of  project  dimensions  and  on  the  overall  project  benefit/cost  to  a  high  degree.  Merit  raters  also  agreed 
highly  on  the  overall  merit  ratings  of  projects,  and  these  two  groups  valued  many  similar  dimensions  associated 
with  the  projects  Due  to  this  high  level  of  agreement  among  the  raters,  confidence  in  the  results  is  high 


The  reliability  of  a  smaller  group  of  raters  can  be  estimated,  but  the  reliability  achieved 
with  a  smaller  set  needs  to  tracked. 

Second,  the  ratings  were  gathered  on  projects  already  active  in  FY86.  Due  to 
intense  review  of  all  projects  active  during  that  particular  fiscal  year  because  of  many 
external  review  committees  visiting  NAVPERSRANDCEN,  many  raters  may  have  had 
more  information  available  to  them  than  would  be  typical.  In  addition,  ratings  of 
historical  data  may  be  much  more  reliable  than  ratings  of  data  gathered  for  proposed 
projects.  Certainly  there  is  more  information  available  on  an  active  project  and  greater 
familiarity  among  the  raters  with  it  than  with  one  that  is  proposed.  Again,  the 
reliability  of  the  process  must  be  tested  using  projects  that  have  not  yet  been  initiated. 

Third,  the  process  at  NAVPERSRANDCEN  included  only  those  projects  that 
were  already  accepted  and  initiated;  it  did  not  include  those  projects  that  were  proposed 
at  the  start  of  the  fiscal  year  but  not  accepted  or  funded.  The  process  needs  to  be  tested 
at  the  beginning  of  a  fiscal  year  and  include  all  proposed  projects.  Then  the  ability  of 
the  equation  to  accurately  predict  merit  under  these  circumstances  can  be  assessed. 

Future  Applications 

This  procedure  has  many  potential  applications  both  within  NAVPERS¬ 
RANDCEN  and  within  other  R&D  organizations.  The  data  collection  forms  were 
revised  to  include  only  that  information  found  to  be  significant  or  thought  to  be 
important  in  the  future  for  NAVPERSRANDCEN  (Appendices  F  and  G).  The  proposed 
rating  form  would  need  to  be  revised  for  each  organization  to  include  dimensions  senior 
managers  believe  to  be  relevant  and  important  for  their  specific  situation.  Instructions 
have  been  included  on  how  to  calculate  predicted  merit  for  proposed  projects  (Appendix 
H). 

There  are  several  issues  to  consider  when  implementing  LAWPAS.  First, 
managers  need  to  determine  which  projects  should  be  assessed  using  the  policy  equation 
Should  all  projects  be  evaluated  or  just  reimbursable  projects  so  that  program  element 
tasks  that  have  been  planned  in  the  funding  cycle  are  exempt  from  the  procedure? 

Second,  there  is  the  issue  of  when  to  apply  the  equation  to  projects.  If  the 
equation  will  be  used  to  assess  the  merit  of  possible  program  element  tasks,  should  these 
be  evaluated  prior  to  submitting  requests  for  funds?  What  about  applying  the  equation 
to  ongoing  projects  to  help  determine  if  resources  are  being  used  in  an  optimal  manner'1 

A  third  issue  in  implementing  LAWPAS  concerns  gaining  an  understanding  of 
the  meaning  of  the  merit  scores.  When  used  over  time,  a  distribution  of  the  predicted 
merit  scores  can  be  constructed.  It  is  then  possible  to  gain  a  clearer  understanding  of 
the  meaning  of  the  predicted  scores.  It  would  be  possible  then  to  define  ranges  or  cut 
scores  based  on  resources  available.  These  could  be  regularly  revised  to  allow  for 
changes  in  the  distribution  of  the  predicted  scores  along  with  changes  in  the  operating 
environment  of  the  organization. 

It  is  important  to  note  that  the  results  from  LAWPAS  capture  policy  viewpoints 
at  a  particular  point  in  time.  For  it  to  be  an  effective  management  tool,  the  process 
should  be  repeated,  possibly  every  year,  so  as  to  accurately  reflect  changes  in  priorities 
and  policy  viewpoints. 

There  are  several  considerations  when  transitioning  LAWPAS  to  new 
organizations.  It  would  be  desirable  to  be  able  to  make  a  quick  assessment  of  an 
organization  to  determine  if  acceptable  agreement  can  be  found  among  managers  in  their 
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ratings  of  projects.  If  a  high  enough  level  of  reliability  in  ratings  is  not  achieved,  there 
are  several  possible  explanations.  The  instruments  used  to  gather  the  ratings  may  not  be 
effective  for  this  particular  organization.  Or  it  may  be  that  there  is  little,  if  any, 
agreement  on  the  values  assigned  to  the  dimensions  of  the  rated  projects.  If  such  a 
situation  exists,  it  may  not  be  worthwhile  to  continue  with  the  LAWPAS  process  unless 
the  organization  wishes  to  do  so  for  self-assessment  purposes,  with  no  expectation  of  a 
reliable  policy  equation  resulting  as  an  end-product. 

Describing  LAWPAS  to  managers  who  lack  an  extensive  understanding  of 
statistics  and  reliability  measures  may  also  be  a  problem  when  transitioning  LAWPAS  to 
other  R&D  organizations.  What  is  a  meaningful  method  for  describing  the  results  of  this 
process  to  managers  and  engineers?  How  else  may  these  data  be  conveyed  so  as  to  make 
sense? 

LAWPAS  proved  to  be  successful  at  NAVPERSRANDCEN.  Managers  felt  that 
the  results  were  useful  in  clarifying  important  dimensions  for  reviewing  potential  R&D 
projects.  The  level  of  agreement  achieved  and  defined  through  the  process  may  lead  to 
a  much  more  systematic  evaluation  and  acceptance  of  work.  It  was  useful  for  senior 
managers  to  realize  that  they  agree  with  each  other  in  their  policy  viewpoints  more  than 
they  disagree.  LAWPAS  also  served  as  a  communication  tool  between  senior  and  middle 
managers,  highlighting  similarities  and  differences  in  their  viewpoints.  It  remains  to  be 
seen  whether  NAVPERSRANDCEN  will  fully  implement  LAWPAS  over  time,  but  as  an 
organizational  assessment  tool,  it  has  served  its  purpose  well. 
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Project  Number 


4/20/87 


Rater _ 

Please  complete  the  following  items  for  the  above  identified  project.  Information  detailing  the 
project  is  attached  for  your  reference.  Reference  to  the  Navy  includes  both  Navy  and  Marine 
Corps.  If  you  are  unable  give  an  exact  response,  then  give  your  best  estimate. 

A.  Attributes  of  the  Problem  Addressed 

1.  To  what  extent  are  the  issues  addressed  by  this  project  important  to: 


a)  NPRDC? 


0  1 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

b)  the  Navy 

R&D  community 

.0 

0  1 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

c)  Government  funding  organizations  (e.g.. 

SYSCOMS,  Congress)? 

0  1 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

d)  The  operational  Navy? 

0  1 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

e)  The  private  sector? 

0  1 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

2.  To  what  extent  do  the  issues  addressed  by  this  project  correspond  with  NPRDC’s  formal 
mission? 

0  1 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

3.  To  what  extent 

is  NPRDC  uniquely  qualified  to  do  this  work? 

0  I 

Not  at  all 

2  3 

4  5 

Somewhat 

6 

7 

8  9 

10 

To  a  great 
extent 

A-l 


4.  To  what  extent  are  other  Navy  R&D  centers/laboratories  better  able  to  handle  this  project 


0  1 
Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


B.  Attributes  of  the  Project 


1.  To  what  extent  does  this  project  build  upon: 

a)  NPRDC  research  and  development? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


b)  other  Navy  research  and  development? 


0  1 
Not  at  all 


4  5  6  7 

Somewhat 


9  10 

To  a  great 
extent 


c)  other  DoD  research  and  development? 


0  1 
Not  at  all 


4  5  6  7 

Somewhat 


9  10 

To  a  great 
extent 


2.  To  what  extent  does  this  project  integrate  with  current: 
a)  NPRDC  projects? 


0  1 
Not  at  all 


4  5  6  7 

Somewhat 


9  10 

To  a  great 
extent 


b)  other  Navy  R&D  projects? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


c)  other  DoD  R&D  projects? 

0  12  3 

Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


3.  To  what  extent  does  the  approach  for  this  project  seem  to  be: 


a)  methodologically  sound" 

0  12  3 

Not  at  all 


4  5  6 

Somew  hat 


9  10 

To  a  great 
extent 


■  A. 


b)  theoretically  sound? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


10 

To  a  great 
extent 


c)  feasible? 


Not  at  all 


4  5  6 

Somewhat 


10 

To  a  great 
extent 


4.  To  what  extent  are  there: 

a)  administrative  risks  (delays,  cuts  in  funding,  etc.)  associated  with  pursuing  this 
project? 


Not  at  all 


4  5  6 

Somewhat 


10 

To  a  great 
extent 


b)  technical  risks  associated  with  pursuing  this  project? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


10 

To  a  great 
extent 


C.  Attributes  of  End-product(s) 

1.  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  to  the  R&D  community' 


Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


2.  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  for  Navy  applications'. 


Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


3.  To  what  extent  will  the  end-product(s)  from  this  project  reflect  positively  on  NPRDC? 


Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


4.  What  is  the  probability  that  end-products  from  this  project  could  be  implemented? 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

5.  To  what  extent  will  this  work  facilitate  future  opportunities  for  funding  and  or  application 


Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 
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6.  Please  indicate  to  what  extent  this  project,  compared  to  other  NPRDC  projects,  is  or  will  be 
important  and  valuable  to  the  Navy  by: 

a)  improving  Navy  effectiveness  (e.g.,  meeting  operational  requirements,  achieving  stated 
goals)? 


0  1 
Value  less 
than  90% 
of  projects 


4  5  6 

Value  equal 
to  middle  20% 
of  projects 


8  9  10 

Value  greater 
than  90% 
of  projects 


b)  improving  Navy  efficiency  (e.g.,  improving  use  of  resources,  reducing  costs)? 
2  3  4  5  6  7  8 


0  I 
Value  less 
than  90% 
of  projects 


4  5  6 

Value  equal 
to  middle  20% 
of  projects 


9  10 

Value  greater 
than  90% 
of  projects 


c)  improving  Navy  personnel  quality  of  life  (e.g.,  improve  morale,  increase  job  satisfaction)? 
2  3  4  5  6  7  8 


0  1 
Value  less 
than  90% 
of  projects 


4  5  6 

Value  equal 
to  middle  20% 
of  projects 


9  10 

Value  greater 
than  90% 
of  projects 


d)  expanding  the  technical  base  (e.g.,  developing  new  technology,  procedures)? 

3  4  5  6  7  8 


0  1 
Value  less 
than  90% 
of  projects 
rojects 


4  5  6 

Value  equal 
to  middle  20% 
of  projects 


9  10 

Value  greater 
than  90% 
of  p 


7.  Overall,  to  what  extent  do  you  perceive  the  benefits  of  undertaking  this  project  outweighing 
the  costs? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


8 


9  10 

To  a  great 
extent 
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APPENDIX  B 

PRINCIPAL  INVESTIGATOR  QUESTIONNAIRE 


PRINCIPAL  INVESTIGATOR  QUESTIONNAIRE 


The  information  you  provide  here  is  part  of  the  information  that  will  used  by  a  set  of  judges  to  rate 
the  attributes  of  selected  projects.  Please  complete  the  following  items  as  accurately  and  completely  as 
possible. 


Project/Work  Unit 


(see  attached  1498) 


PI 


1.  Please  check  the  organizations  funding  and/or  endorsing  this  project.  For  each  organization 
checked,  indicate  the  specific  code  providing  funding  and/or  endorsing  the  project. 

SECNAV 

ONT 

ASN 

ONR 

DoD 

NAVSEA 

OPNAV 

NAVAIR 

SPAWAR 

NAVSUP 

NMPC 

NAVDAC 

NCPC 

NAVFAC 

NRC 

Other  (specify) 

CNR 

Other  (specif y) 

CNET 

Other  (specify) 

CN’TT 

2.  Please  indicate  the  rank  or  grade  of  the  highest  official  endorsing  this  project. 


Political  Appointee 

Elected  Official 

0-10 

0-9 

0-8  0-7 

0-6  0-5 

SES 

GM  GS-15 

GM/GS-14 

GM  GS-13 

3.  What  funding  level  (SK)  was  originally  planned  for  this  project  over  its  entire  duration0 


4.  Was 'is  this  project  part  of  an  integrated  set  of  projects? 

If  so,  what  percent  of  the  total  effort  does  it  represent? 

0%  10%  20%  30%  40%  50%  60%  70%  80%.  90%  100% 

5.  To  what  extent  does  this  work  support  elements  of  your  department's  Work  Breakdown  Structure0 

0123456789  10 

Not  at  all  Somewhat  To  a  great 

extent 


6.  a)  What  was  is  the  planned  start  date  of  this  project0 

b)  What  was  the  actual  start  date'.’ 

c)  What  was/ is  the  planned  end  date  ot  this  project0 

d)  If  completed,  what  was  the  actual  end  date0 


7.  What  percent  of  the  in-house  resources  (e.g.,  personnel,  equipment,  facilities)  required  to  perform 
this  project  were  available  when  the  project  was  initiated? 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


8.  If  less  than  100%,  how  difficult  was  it  to  obtain  the  remaining  resources? 


0 

Not  at  all 
difficult 


Somewhat 

difficult 


9  10 

Extremely 
difficult 


9.  What  percent  of  the  total  funding  for  this  project  was  contracted? 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


10.  Please  indicate  the  number  of  end-products  this  project  has  or  is  planned  to  produce  over  its 
entire  duration  for  each  category: 

_ Sof  tware 

_ Hardware 

_ Operating/administrative  procedures 

_ System/equipment  design  specifications 

_ NPRDC  technical  note/report 

_ Journal  article 

_ Briefing 

_ Convention  paper 

_ _Other  (specify) _ 

11.  Please  indicate  what  did/will  change  as  a  result  of  this  project’s  end-products. 

_ Information/knowledge 

_ Policy 

_ Procedures/Techniques 

12.  Please  check  the  setting(s)  in  which  end-products  of  this  project  are,  will  be  used. 

_ The  operational  fleet 

_ Navy  shore  activities 

_ Navy  industrial  activities 

_ Navy  R&D  centers  and  laboratories 

_ Other  military  activities 

_ Government  or  DoD  headquarters  (policy  makers) 

_ Universities 

Private  sector  businesses 


Other  (specify) 


13.  To  what  extent  did/will  each  of  the  following  groups  be  directly  affected  by  this  project's  end- 
products: 


a)  Military  recruits 


0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

b)  Military  officers 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

c)  Enlisted  personnel 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

d)  Military  dependents 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

e)  Government  civilian  employees 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

f)  Government  contractors 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

g)  Other  private  sector  personnel 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

14.  How  many  months  did/will  elapse  from  the  initiation  of  this  project  until  end-products  were  will 

be  delivered  to  a  customer?  _ months 

15.  To  what  extent  did/will  existing  conditions  have  to  be  altered  to  implement  end-products  from  th 
project? 

0123456789  10 

Not  at  all  Somewhat  To  a  great 

extent 


APPENDIX  C 


SAMPLE  OF  FORM  1498 


WWW 


wvww 


RESEARCH  AND  TECHNOLOGY  WORK  UNIT  SUMMARY 


4 

CM  D  OF 

5  Summary 

SUMMAR  v 

sc  -  » 

D 

CHANGE 

u 

1  AGENCv  ACCESSION 

DN297508 


7  REGRADING 


8  DlSB'N  INSTR  N 

cx 


b  CONTRlBUT.NG 


c  CONTRIBUTING 


3  I  PERSONNEL  AND  TRAINING  |  _ 


11  TiTlE  (Precede  u.ith  Securit.  Classification  Coae>  (U^  PERSONNEL  ASSIGNMENT 


12  SuBJ6C~  AREAS 


13  start  date 

01  OCT  80 


17  CONT  R  ACT.  GR  AN  T 


a  OATe  ELECTIVE 


14  ESTIMATED  COMPLETION  date 

30  SEP  87 


6  PERFORMANCE  method 

C  IN-HOUSE 


18  RESOURCES  ESTIMATE 


FISCAL  YEARS  a.  PROFESSIONAL  WORK  vEAP 


K  NO  CF  AWA«C 


RESPCNSiBuE  DC  C 


I 

E 

fl 


NAVY  PERSONNEL  RESEARCH  AND 
DEVELOPMENT  CENTER _ 

DDR6SS  ‘include  zip  cod. 

SAN  DIEGO,  CA  92I52-66CC 

JM£  OF  B[SFC\S.St£  INC  V'DLSL 

LAN'CO  .  T  .  A  . _ 6 

El£phON£  NUMBEc  'mc.uiJ.  area  code 


a  name  NAVY  PERSONNEL  RE 
DEVELOPMENT  CENT E 


D  ADDRESS 

SAN  DIEGO,  CA  92152-fccC: 

c  NAME  OF  PR. NCIPAL  INVES*  GA*OFi 

LIANG.  T. 


a  TELEPHONE  NUMBER  (include  arv a  eva, 

619-225-237 • 


21  GENE  SAL  USE 

H  M'L  T  a  P  y  C'V  L’AN  APP  L'CA^CN 


3  TEC^NiCAl  CE.fcC'-V'E  24  appro  AC1-*  25  PROGRESS  -Precede  text  of  each  u.tth  Security  Classification  Cooe 

23.  OBJECTIVE:  <U)  NAVY  PERSONNEL  ASSIGNMENT  IS  A  LARGE,  COMPLEX 
OPERATION.  OVER  500,000  ACTIVE  ENLISTED  PERSONNEL  ARE  ASSIGNED  TO  SOM 
5,000  DIFFERENT  ACTIVITIES  CR  UNITS  IN  ABOUT  300  DISTINCT  OCC’J? A  .  IONA 
SPECIALTIES  AND  AT  NINE  DIFFERENT  SKILL  LEVELS.  THE  OBJECTIVE  Or  THIS 
EFFORT  IS  TO  INVESTIGATE  AND  DEVELOF  METHODOLOGIES  TO  5GLVE  COMFLEX, 
LARGE-SCALE,  MULTIPLE-CRITERIA  OPTIMIZATION  PROBLEMS  ASSOCIATED  WITH 
NAVY  PERSONNEL  ASSIGNMENT.  FORMER  WORK  UNIT  NUMBER  521-604-C31 .  FORME 
TITLE  CAREER  MANAGEMENT  PLANNING  (CAMP).  INCLUDES  WORK  UNIT  WR-22025. 

24.  APPROACH:  <U)  THE  APPROACH  IS  TO  (1)  INVESTIGATE  AND  DEVELOP  NEW 
COMPUTATIONAL  NETWORK  CODES  CAPABLE  OF  HANDLING  A  LARGE  VARIETY  CF 
OBJECTIVES  AND  CONSTRAINTS  FOR  THE  NAVY'S  ASSIGNMENT  PROBLEM,  (2) 
EXPLORE  STRUCTURED  MODELING  AND  AGGREGATION  THEORY  FOR  INTERFACING  TH 
LINKS  IN  POLICY  SETTING  AT  THE  STRATEGIC  LEVEL  AND  IMPLEMENTATION  AT 
OPERATIONAL  LEVEL,  AND  (3)  EXAMINE  THE  POTENTIAL  USE  OF  ARTIFICIAL 

I NTELL I GENCE /EXPERT  SYSTEMS  TECHNOLOGY  IN  ASSIGNMENT  SYSTEMS. 

25.  PR0GRE5S :  (U)  (OCT  65  -  MAR  66):  A  NEW  COMPUTATIONAL  NETWORK  CODE 
WAS  DEVELOPED  THAT  CAN  SOLVE  AN  ASSIGNMENT  PROBLEM  WITH  19  ENROUTE 
TRAINING  CAPACITY  CONSTRAINTS  THAT  ARE  UNIFORM  (I.E.,  THE  0UOT AS  ARE 
THE  5AME ) .  THIS  DEVELOPMENT  IS  AN  ENCOURAGING  STEF  TOWARD  DEVELOPING 
ALGORITHMS  TO  SOLVE  ASSIGNMENT  PROBLEMS  FOR  ENLISTED  RATINGS  WITH 
MULTIPLE  NAVY  ENLISTED  CLASSIFICATIONS  AND  CLASS  'C'  SCHOOLS  THAT  MI’S 
UNDERGO  EXTENSIVE  ENROUTE  TRAINING. 

i  o  o  Navy  Personnel  flesea'cM  ana  Development  Cente* 


1498  NPPDC  .OP  3500  1  R 


Table  D- 1 


MEANS.  STANDARD  DEVIATIONS,  AND  RELIABILITY  FOR  VARIABLES 

(Cases  =  60) 


VARIABLE 

MEAN 

STD  DEV 

RELIABILITY 

MERIT 

6.000 

1.297 

.86 

B  C 

6.226 

1.445 

.92 

MISCNTRD 

5.682 

1.421 

.91 

NAVYOP 

6.005 

1.482 

.89 

RISK 

3.929 

1.944 

.84 

DODRD 

3.634 

1.597 

.88 

TRANSITN 

6.898 

1.210 

.77 

SOUND 

6.787 

1.294 

7  ^ 

ACTFUND 

202445.000 

522.554 

LFUND 

5.086 

.476 

TYPE1 

.033 

.181 

TY pe2 

.200 

.403 

TYPE3 

.233 

.427 

TYPE4 

.017 

.129 

T'i  PER 

.483 

.504 

INTEGRTN 

.533 

.503 

PERCENT 

64.600 

36.392 

YRS 

2.817 

1.935 

INHOUSE 

74.500 

28.71 1 

DIFFCLTY 

2.867 

3.100 

CONTRCT% 

23.383 

21.381 

INF/ KNOW 

.833 

.376 

POLICY 

.383 

.490 

PROCEDR 

.850 

.360 

SPONSORS 

1.800 

1.162 

SETTINGS 

2.433 

1.671 

Table  D-2 


MULTIPLE  REGRESSION  ON  MERIT  RATERS’ 
AVERAGE  OVERALL  MERIT  RATINGS  (MERIT) 

DEPENDENT  VARIABLE  MERIT 

MULTIPLE  R  .83005 

R  SQUARE  .68898 

ADJUSTED  R  SQUARE  .66018 

STANDARD  ERROR  .75598 

ANALYSIS  OF  VARIANCE 


DF 

SUM  OF  SQUARES  MEAN  SQUARE 

REGRESSION 

5 

68.36427 

13.67285 

RESIDUAL 

54 

30.861 13 

.57150 

F  =  23.9:441 

S1GN1F  F  =  .0000 

VARIABLES  IN  THE  EQUATION 


variable 

B 

SE  B 

BETA 

T 

SIC.  T 

MISCNTRD 

.630101 

.080469 

.690408 

7.830 

.0000 

LFUND 

.872882 

.247518 

.320504 

3.527 

.0009 

DODRD 

-.196120 

.070371 

-.241561 

-2.787 

.0073 

POLICY 

-.559802 

.211184 

-.211647 

-2.651 

.0105 

TYPE3 

.524012 

.250030 

.172344 

2.096 

.0408 

(CONSTANT) 

-1.214168 

1.122480 

-1.082 

.2842 

Table  D-3 


MULTIPLE  REGRESSION  ON  ATTRIBUTE  RATERS’ 
AVERAGE  OVERALL  BENEFIT/COST  RATINGS  (B/C) 

DEPENDENT  VARIABLE  B/C 

MULTIPLE  R  .92119 

R  SQUARE  .84858 

ADJUSTED  R  SQUARE  .83456 

STANDARD  ERROR  .58776 


ANALYSIS  OF  VARIANCE 


DF 

SUM  OF  SQUARES  MEAN  SQUARE 

REGRESSION- 

5 

104.52705  20.90541 

RESIDUAL 

54 

18.65119  .34539 

F  =  60.52654 

SIGNIF  F  =  .0000 

VARIABLES  IN  THE  EQUATION 


VARIABLE 

B 

SE  B 

BETA 

T 

SIG  T 

M1SCNTRD 

.785814 

.055213 

.772787 

14.232 

.0000 

TRANSITN 

.451882 

.066938 

.378442 

6.751 

.0000 

TYPE3 

.500490 

.185194 

.147739 

2.703 

.0092 

RISK 

-.114400 

.042347 

-.153950 

-2.702 

.0092 

INHOUSE 

-.007193 

.002768 

-.142938 

-2.598 

.0120 

(CONSTANT) 

-.487197 

.662315 

-.736 

.4652 

\v.v,-\v^  A'y^V.VV,\'  v.m\’ '•■* VVvrV  '.  «•  v-;  w-".— .•-. 
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CORRELATION  MATRIX 


MERIT 

B/C 

MISCNTRD 

NAVYOP 

RISK 

DODRD 

TRANSITS 

MERIT 

1. 000 

.550 

.718 

.553 

.218 

.165 

.001 

B  C 

.550 

1.000 

.758 

.721 

-.133 

.259 

.444 

M1SCNTRD 

.718 

.758 

1.000 

.626 

.166 

.433 

.004 

NA VVOP 

.553 

.721 

.626 

1.000 

-.096 

.308 

.345 

RISK 

.218 

-.133 

.166 

-.096 

1.000 

.152 

-.321 

DODRD 

.165 

.259 

.433 

.308 

.152 

1.000 

-.177 

TRANSITS’ 

.001 

.444 

.004 

.345 

-.321 

-.177 

1.000 

SOUND 

.242 

.591 

.399 

.433 

-.335 

.106 

.406 

TVPE1 

.068 

.244 

.203 

.070 

-.152 

.081 

.022 

TYPE2 

.212 

.048 

.245 

.090 

.073 

.249 

-.179 

TVPE3 

.300 

.235 

.067 

.220 

.070 

-.044 

.045 

TVPE4 

.098 

-.006 

.078 

-.009 

.030 

.092 

-.034 

TYPER 

-.474 

-.323 

-.346 

-.283 

-.073 

-.216 

.108 

ACTFUND 

.464 

.176 

.327 

.290 

.223 

.241 

.003 

LFUND 

.528 

.231 

.396 

.329 

.213 

.305 

-.027 

INTEGRTN 

.091 

.014 

.258 

-.115 

.298 

.334 

-.320 

PERCENT 

-.062 

.008 

-.284 

.069 

-.301 

-.350 

.374 

YRS 

.477 

.173 

.307 

.291 

.228 

.130 

-.161 

INHOUSE 

-.156 

-.304 

-.164 

-.054 

-.018 

.076 

-.107 

DIFFCLTY 

.280 

.291 

.195 

.024 

.074 

.021 

.047 

CONTRCT% 

.170 

.070 

.169 

.170 

.041 

.218 

.094 

INF/KNOW 

.319 

.201 

.242 

-.026 

.137 

-.047 

-.049 

POLICY 

-.057 

-.003 

.051 

.108 

-.064 

-.048 

-.009 

PROCEDR 

.048 

.089 

.152 

.106 

.069 

.092 

-.066 

SPONSORS 

.291 

.064 

.213 

.024 

.149 

.094 

-.083 

SETTINGS 

.033 

.124 

.243 

.107 

-.065 

.317 

-.130 

s  •.  *.  **. 
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CORRELATION  MATRIX  (CON’T) 


SOUND 

TYPE1 

TYPE2 

TYPE3 

TYPE4 

TYPER 

ACTFUND 

MERIT 

.242 

.068 

.212 

.300 

.098 

-.474 

.464 

B/C 

.591 

.244 

.048 

.235 

-.006 

-.323 

.176 

M1SCNTRD 

.399 

.203 

.245 

.067 

.078 

-.346 

.327 

NAVYOP 

.433 

.070 

.090 

.220 

-.009 

-.283 

.290 

RISK 

-.335 

-.152 

.073 

.070 

.030 

-.073 

.223 

DODRD 

.106 

.081 

.249 

-.044 

.092 

-.216 

.241 

TRANSITN 

.406 

.022 

-.179 

.045 

-.034 

.108 

.003 

SOUND 

1.000 

.302 

-.053 

.141 

.074 

-.201 

.197 

TYPE  I 

.302 

1.000 

-.092 

-.101 

-.023 

-.184 

-.119 

TYPE2 

-.053 

-.092 

1.000 

-.285 

-.065 

-.518 

-.077 

TYPE3 

.141 

-.101 

-.285 

1.000 

-.071 

-.568 

TYPE4 

.074 

-.023 

-.065 

-.071 

1.000 

-.129 

.105 

TYPER 

-.201 

-.184 

-.518 

-.568 

-.129 

1.000 

-.108 

ACTFUND 

.197 

-.119 

-.077 

.222 

.105 

-.108 

1.000 

LFUND 

.194 

-.126 

.056 

.349 

.134 

-.327 

.778 

INTEGRTN 

.056 

-.014 

.235 

-.007 

.117 

-.208 

.210 

PERCENT 

.018 

.053 

-.272 

.168 

-.157 

.098 

-.104 

YRS 

.147 

.020 

.241 

.335 

.080 

-.516 

.357 

INHOUSE 

-.258 

-.085 

.157 

-.175 

-.017 

.061 

-.052 

DIFFCLTY 

.271 

.124 

-.175 

.253 

-.036 

-.113 

.207 

CONTRCT% 

.054 

-.078 

.032 

.092 

.235 

-.138 

.258 

INF/KNOW 

.100 

.079 

.114 

.141 

.055 

-.254 

.196 

POLICY 

.093 

-.147 

-.093 

.156 

-.103 

.020 

.240 

PROCEDR 

.084 

.074 

.208 

.015 

.052 

-.220 

.096 

SPONSORS 

-.032 

-.129 

-.087 

.061 

.244 

.003 

.564 

SETTINGS 

.165 

.210 

.142 

-.084 

.035 

-.126 

.115 

r 

i 

! 

i 
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CORRELATION  MATRIX  (CON'T) 


LFUND 

INTEGRTN 

PERCENT 

YRS 

INHOUSE 

DIFFCLTY 

CONTRCT 

MERIT 

.528 

.091 

-.062 

.477 

-.156 

.280 

.170 

B/C 

.231 

.014 

.008 

.173 

-.304 

.291 

.070 

MISCNTRD 

.396 

.258 

-.284 

.307 

-.164 

.195 

.169 

NAVYOP 

.329 

-.115 

.069 

.291 

-.054 

.024 

.170 

RISK 

.213 

.298 

-.301 

.228 

-.018 

.074 

.041 

DODRD 

.305 

.334 

-.350 

.130 

.076 

.021 

.218 

TRANSITN 

-.027 

-.320 

.374 

-.161 

-.107 

.047 

.094 

SOUND 

.194 

.056 

.018 

.147 

-.258 

.271 

.054 

TYPE  1 

-.126 

-.014 

.053 

.020 

-.085 

.124 

-.078 

TYPE2 

.056 

.235 

-.272 

.241 

.157 

-.175 

.032 

TYPE3 

.349 

-.007 

.168 

.335 

-.175 

.253 

.092 

TYPE4 

.134 

.117 

-.157 

.080 

-.017 

-.036 

.235 

TYPER 

-.327 

-.208 

.098 

-.516 

.061 

-.113 

-.138 

ACTFUND 

.778 

.210 

-.104 

.357 

-.052 

.207 

.258 

LFUND 

1.000 

.239 

-.120 

.420 

-.056 

.148 

.267 

INTEGRTN 

.239 

1.000 

-.91 1 

.290 

.003 

.087 

.151 

PERCENT 

-.120 

-.911 

1.000 

-  194 

-.007 

-.049 

-.130 

YRS 

.420 

.290 

-.194 

1.000 

-.088 

.173 

-.102 

INHOUSE 

-.056 

.003 

-.007 

-.088 

1.000 

-.857 

-.086 

DIFFCLTY 

.148 

.087 

-.049 

.173 

-.857 

1.000 

.044 

CONTRCT% 

.267 

.151 

-.130 

-.102 

-.086 

.044 

1.000 

INF/ KNOW 

.449 

.209 

-.138 

.278 

-.357 

.374 

.080 

POLICY 

.236 

.033 

-.089 

.138 

.069 

-.032 

-.118 

PROCEDR 

.047 

.260 

-.226 

.155 

.208 

-.178 

.149 

SPONSORS 

.392 

.181 

-.144 

.046 

-.127 

.113 

.322 

SETTINGS 

.183 

.302 

-.308 

.110 

-.070 

.110 

.094 

.094 


CORRELATION  MATRIX  (CON’T) 


INF/KNOW 

POLICY 

PROCEDR 

SPONSORS 

SETTINGS 

MERIT 

.319 

-.057 

.048 

.291 

.033 

B/C 

.201 

-.003 

.089 

.064 

.124 

MISCNTRD 

.242 

.051 

.152 

.213 

.243 

NAVYOP 

-.026 

.108 

.106 

.024 

.107 

RISK 

.137 

-.064 

.069 

.149 

-.065 

DODRD 

-.047 

-.048 

.092 

.094 

.317 

TRANSITN 

-.049 

-.009 

-.066 

-.083 

-.130 

SOUND 

.100 

.093 

.084 

-.032 

.165 

TYPE1 

.079 

-.147 

.074 

-.129 

.210 

TYPE2 

.114 

-.093 

.208 

-.087 

.142 

TYPE3 

.141 

.156 

.015 

.061 

-.084 

TYPE4 

.055 

-.103 

.052 

.244 

.035 

TYPER 

-.254 

.020 

-.220 

.003 

-.126 

ACTFUND 

.196 

.240 

.096 

.564 

.115 

LFUND 

.449 

.236 

.047 

.392 

.183 

INTEGRTN 

.209 

.033 

.260 

.181 

.302 

PERCENT 

-.138 

-.089 

-.226 

-.144 

-.308 

YRS 

.278 

.138 

.155 

.046 

.110 

INHOUSE 

-.357 

.069 

.208 

-.127 

-.070 

DIFFCLTY 

.374 

-.032 

-.178 

.113 

.110 

CONTRCT% 

.080 

-.118 

.149 

.322 

.094 

INF/KNOW 

1.000 

.086 

-.053 

.232 

.210 

POLICY 

.086 

1.000 

.053 

.022 

.188 

PROCEDR 

-.053 

.053 

1.000 

-.068 

.099 

SPONSORS 

.232 

.022 

-.068 

1.000 

.125 

SETTINGS 

.210 

.188 

.099 

.125 

1.000 

PROJECT  INFORMATION  SHEET 


The  information  you  provide  here  is  part  of  the  information  that  will  used  by  a  set  of  judges  to  rate 
your  project.  Please  complete  the  following  items  as  accurately  and  completely  as  possible. 

Project  Title _ 

PI _ code _ phone _ 

1.  Please  check  the  organizations  funding  and/or  endorsing  this  project.  For  each  organization 
checked,  indicate  the  specific  code  providing  funding  and/or  endorsing  the  project. 

_ SECNAV 

_ ASN _ 

_ DoD _ 

_ OPNA V_ 

_ SPA  WAR 

_ NMPC _ 

_ NCPC _ 

_ NRC _ 

_ CNR _ 

_ CNET _ 

_ CNTT _ 

2.  (a)  What  is  the  expected  funding  level  ($K)  for  this  project  this  year? _ 

(b)  What  is  the  expected  funding  level  over  the  life  of  the  project? _ 

3.  Is  this  project  part  of  an  integrated  set  of  projects? _ 

If  so.  what  percent  of  the  total  effort  does  it  represent? 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

4.  To  what  extent  does  this  work  support  elements  of  your  department’s  Work  Breakdown  Structure? 

0123456789  10 

Not  at  all  Somewhat  To  a  great 

extent 

5.  Which  element  of  the  Work  Breakdown  Structure  does  this  project  most  closely  fit? 

6.  a)  What  is  the  planned  start  date  of  this  project? _ 

b)  W'hat  is  the  planned  end  date  of  this  project? _ 

7.  What  percent  of  the  in-house  resources  (e.g.,  personnel,  equipment,  facilities)  required  to  perform 
this  project  will  be  available  when  the  project  is  initiated ? 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

8.  What  percent  of  the  total  funding  for  this  project  is  to  be  contracted? 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

9.  Please  indicate  what  will  change  as  a  result  of  this  project's  end-products. 

_ Information/knowledge 

_ Policy 

_ Procedures/Techniques 


ONT _ 

ONR _ 

navsea _ 

NAVAIR _ 

NAVSUP _ 

NAVDAC _ 

NAVFAC _ 

Other  (specify) 
Other  (specify) 
Other  (specify) 


Please  complete  the  following  items  for  the  above  identified  project.  Information  detailing  the 
project  is  attached  for  your  reference.  Reference  to  the  Navy  includes  both  Navy  and  Marine 
Corps.  If  you  are  unable  give  an  exact  response,  then  give  your  best  estimate. 


1.  To  what  extent  are  the  issues  addressed  by  this  project  important  to: 
a)  NPRDC? 


0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

b)  The  operational  Navy? 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

c)  The  private  sector? 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

2.  To  what  extent  do  the  issues  addressed  by  this  project  correspond  w 
mission? 

ith  NPRDC’s  formal 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 
extent 

3.  To  what  extent  is  NPRDC  uniquely  qualified  to  do  this  work? 

0  12  3 

Not  at  all 

4  5 

Somewhat 

6 

7 

8 

9 

10 

To  a  great 

extent 
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4.  To  what  extent  does  this  project  build  upon: 


a)  NPRDC  research  and  development? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


b)  other  Navy  research  and  development? 


Not  at  all 


4  5  6 

Somewhat 


c)  other  DoD  research  and  development? 


Not  at  all 


4  5  6 

Somewhat 


5.  To  what  extent  does  this  project  integrate  with  current: 


a)  NPRDC  projects'- 


0  1 
Not  at  all 


4  5  6  7 

Somewhat 


b)  other  Navy  R&D  projects? 


0  1 
Not  at  all 


4  5  6  7 

Somewhat 


c)  other  DoD  R&D  projects? 


012345678 
Not  at  all  Somewhat 


6.  To  what  extent  does  the  approach  for  this  project  seem  to  be: 


a)  methodologically  sound? 


012345678 
Not  at  all  Somewhat 


b)  theoretically  sound? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


c)  feasible? 


0  1 
Not  at  all 


4  5  6 

Somewhat 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


9  10 

To  a  great 
extent 


7.  To  what  extent  are  there: 


a)  administrative  risks  (delays,  cuts  in  funding,  etc.)  associated  with  pursuing  this 
project? 


0  1 
Not  at  all 


23456789  10 

Somewhat  To  a  great 


extent 


b)  technical  risks  associated  with  pursuing  this  project? 


0  1 
Not  at  all 


23456789  10 

Somewhat  To  a  great 

extent 


8.  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  to  the  R&D  community 


0  1 
Not  at  all 


23456789  10 

Somewhat  To  a  great 

extent 


9.  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  for  Navy  applications'! 


0  1 
Not  at  all 


23456789  10 

Somewhat  To  a  great 

extent 


10.  To  what  extent  will  the  end-product(s)  from  this  project  reflect  positively  on  NPRDC? 


0  1 
Not  at  all 


23456789  10 

Somewhat  To  a  great 

extent 


11.  What  is  the  probability  that  end-products  from  this  project  could  be  implemented? 
0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


12.  To  what  extent  will  this  work  facilitate  future  opportunities  for  funding  and/or 
applications? 


0  1 
Not  at  all 


23456789  10 

Somewhat  To  a  great 

extent 
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13.  Please  indicate  to  what  extent  this  project,  compared  to  other  NPRDC  projects,  is  or  will 
be  important  and  valuable  to  the  Navy  by: 

a)  improving  Navy  effectiveness  (e.g.,  meeting  operational  requirements,  achieving  stated 
goals)? 


0 

Value  less 
than  90% 
of  projects 


4  5  6 

Value  equal 
to  middle  20% 
of  projects 


9  10 

Value  greater 
than  90% 
of  projects 


b)  improving  Navy  efficiency  (e.g.,  improving  use  of  resources,  reducing  costs)? 


0 

Value  less 
than  90% 
of  projects 


4  5  6 

Value  equal 
to  middle  20% 
of  projects 


8  9  10 

Value  greater 
than  90% 
of  projects 


c)  expanding  the  technical  base  (e.g.,  developing  new  technology,  procedures)? 


0123456789  10 

Value  less  Value  equal  Value  greater 

than  90%  to  middle  20%  than  90% 

of  projects  of  projects  of  projects 

14.  Overall,  to  what  extent  do  you  perceive  the  benefits  of  undertaking  this  project 
outweighing  the  costs? 

0123456789  10 

Not  at  all  Somewhat  To  a  great 

extent 
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APPENDIX  H 

INSTRUCTIONS  FOR  CALCULATING  PREDICTED  MERIT 
FOR  PROPOSED  PROJECTS 


H-0 


LABORATORY  WORKLOAD  PRIORITIZATION  AND  ACCEPTANCE  SYSTEM  (LAWPAS) 


The  following  instructions  describe  how  to  use  LAWPAS  to  predict  the  merit  of  potential 
projects. 

1.  Ask  the  principal  investigator  (PI)  of  the  proposed  project  to  complete  the  Proposed  Project 
Information  Sheet  (Appendix  F)  and  to  provide  any  additional  documentation  describing  the 
project,  such  as  a  project  plan  or  technical  development  plan. 

2.  Identify  three  attribute  raters.  Provide  the  raters  with  the  Proposed  Project  Information 
Sheet  and  documentation  for  the  project.  Ask  the  raters  to  review  these  items  and  to  complete 
the  Proposed  Project  Rating  Form  (Appendix  G). 

3.  Combine  the  three  attribute  rater  scores  for  each  question  and  calculate  the  mean  for  each 
item  in  the  questionnaire.  This  is  the  rating  value  that  will  be  used  in  the  next  step. 

4.  Combine  the  rating  values  from  step  3  to  create  the  variables  in  the  predicted  merit 
equation.  After  totaling  the  appropriate  items,  calculate  the  mean.  This  is  the  value  you  will 
enter  into  the  equation  to  calculate  the  project’s  predicted  merit.  The  questionnaire  items  are 
combined  from  the  Proposed  Project  Rating  Form  (Appendix  G)  as  follows: 

MISCNTRD  Centrality  to  NPRDC  Mission 

Questionnaire  items 

la.  To  what  extent  are  the  issues  addressed  by  this  project  important  to 
NPRDC? 

2.  To  what  extent  do  the  issues  addressed  by  this  project  correspond  with 
NPRDC’s  formal  mission? 

4a.  To  what  extent  does  this  project  build  upon  NPRDC  research  and 
development? 

5a.  To  what  extent  does  this  project  integrate  with  current  NPRDC 
projects? 

8.  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  to 
the  R&D  community? 

10.  To  what  extent  will  the  end-product(s)  from  this  project  reflect 
positively  on  NPRDC? 

12.  To  what  extent  will  this  work  facilitate  future  opportunities  for 
funding  and/or  applications? 

13c.  To  what  extent  is/will  this  project,  compared  to  other  NPRDC 
projects,  be  important  and  valuable  to  the  Navy  by  expanding  the 
technical  base  (developing  new  technology,  procedures,  etc)? 

RISK  Project  Risk 

Questionnaire  items 

3.  To  what  extent  is  NPRDC  uniquely  qualified  to  do  this  work? 

(This  item  is  reverse  scored.) 

4b.  To  what  extent  does  this  project  build  upon  other  Navy  research  and 
development? 

5b.  To  what  extent  does  this  project  integrate  with  current  Navy  R&D 
projects? 


7a.  To  what  extent  are  there  administrative  risks  associated  with  pursuing 
this  project? 

7b.  To  what  extent  are  there  technical  risks  associated  with  pursuing  this 
project? 

DODRD  Connectivity  with  DOD  R&D 

Questionnaire  items 

4c.  To  what  extent  does  this  project  build  upon  other  DoD  research  and 
development? 

5c.  To  what  extent  does  this  project  integrate  with  current  Dod  R&D 
projects? 

TRANSITN  Likelihood  of  Transitioning 
Questionnaire  items 

6c.  To  what  extent  does  the  approach  for  this  project  seem  to  be 
feasible? 

11.  What  is  the  probability  that  end-products  from  this  project  could  be 
implemented? 

Note:  The  following  two  variables  did  not  enter  into  the  current  merit 
equation,  but  may  be  of  interest  to  calculate  and  track. 

NAVYOP  Usefulness  to  Operational  Navy 

Questionnaire  items 

lb.  To  what  extent  are  the  issues  addressed  by  this  project  important  to 
the  operational  Navy? 

9.  To  what  extent  will  the  end-product(s)  from  this  project  be  useful  for 
Navy  applications? 

1 3a.  To  what  extent  is/will  this  project,  compared  to  other  NPRDC 
projects,  be  important  and  valuable  to  the  Navy  by  improving  Navy 
effectiveness  (e.g.,  meeting  operational  requirements,  achieving  stated 
goals,  etc.)? 

13b.  To  what  extent  is/will  this  project,  compared  to  other  NPRDC 
projects,  be  important  and  valuable  to  the  Navy  by  improving  Navy 
efficiency  (e.g.,  improving  use  of  resources,  reducing  costs,  etc.)? 

SOUND  Scientific  validity 

Questionnaire  items 

6a.  To  what  extend  does  the  approach  for  this  project  seem  to  be 
methodologically  sound? 

6b.  To  what  extend  does  the  approach  for  this  project  seem  to  be 
theoretically  sound? 

5.  Gather  the  funding  information  from  Form  1498  for  the  following  variables: 

LFUND  Log  transform  of  the  best  estimate  of  actual  funding  for  FY88  (item  17d 

on  Form  1498  or  item  2a  on  Proposed  Project  Information  Sheet  i. 

TYPE3  If  program  element  funding  type  is  6.3,  value  is  1,  otherwise  0  (item 

10a). 


6.  Gather  the  PI  information  from  the  Proposed  Project  Information  Sheet  for  the  following 
variables: 

INHOUSE  What  percent  of  in-house  resources  required  are  available  at  startup 
(question  7)? 

POLICY  Will  the  end-products  change  policy  (question  9)?  If  true,  value  is  1, 

otherwise  0. 

7.  Put  the  data  from  steps  four  and  five  into  the  following  equation  to  calculate  the  project's 
predicted  merit: 


PMERIT  =  (.630  *  MISCNTRD)  +  (.830  *  LFUND)  +  (-.196  *  DODRD) 

+  (-.560  *  POLICY)  +  (.524  *  Type3)  -  1.214* 

*Notc:  The  weights  reported  here  in  the  equation  differ  from  those  found  in  the  text  because 
they  are  the  unstandardized  regression  weights. 

8.  If  you  are  interested  in  calculating  the  predicted  overall  benefit/cost  score  of  the  project 
(determined  from  attribute  raters’  data),  use  the  following  equation: 

PB/C  =  (.786  *  MISCNTRD)  +  (.452  *  TRANSITN)  +  (.500  *  TYPE 3) 

+  (.007  ♦  INHOUSE)  +  (-.108  *  RISK)  -  .487* 

*Note:  The  weights  reported  here  in  the  equation  differ  from  those  found  in  the  text  because 
they  are  the  unstandardized  regression  weights. 

9.  Compare  the  predicted  score  with  scores  of  existing  projects  and  other  proposed  projects 
to  get  an  indication  of  where  this  project  stands  in  terms  of  predicted  merit. 
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